To repair or not to repair: reconciling correctness and coherence in ontology reference alignments

A recent development in the field of ontology matching is the alignment repair process, whereby mappings that lead to unsatisfiable classes are removed to ensure that the final alignment is coherent. This process was showcased in the Large Biomedical Ontologies track of OAEI 2012, where two repair systems (ALCOMO and LogMap) were used to create separate coherent reference alignments from the original alignment based on the UMLS metathesaurus. In 2013, the OAEI introduced new reference alignments for this track, created by using the two repair systems in conjunction and manual curation when necessary. In this paper, we present the results of a manual analysis of the OAEI 2013 Large Biomedical Ontologies reference alignments, focused on evaluating the equivalence mappings removed by the repair process as well as those that were replaced by subsumption mappings. We found that up to two-thirds of the removed mappings were correct and that over 90% of the analyzed subsumption mappings were incorrect, since in most cases the correct type of relation was the original equivalence. We discuss the impact that disregarding correctness to ensure coherence can have on practical ontology matching applications, as well as on the evaluation of ontology matching systems.